To this point, we haven’t really talked about how we should organize a repository. A repository is just a set of files to track over time, but how do we organize those files?

On some level, this is to be expected with data analysis/data science - we rarely work in a linear progression with the same set of files from project to project.

CookieCutter Data Science

It’s no secret that good analyses are often the result of very scattershot and serendipitous explorations. Tentative experiments and rapidly testing approaches that might not work out are all part of the process for getting to the good stuff, and there is no magic bullet to turn data exploration into a simple, linear progression.

It’s less important to have the perfect organization for a given project than it is to have some sort of standard that everyone understands and uses.

Sadly, this template is intended for Python, but we can adapt it for R easily enough. Let’s zoom in a bit on specific pieces.

├── Makefile        <- Makefile with commands like `make data` or `make train`
├── README.md       <- The top-level README for developers
├── data
│   ├── external    <- Data from third party sources.
│   ├── interim     <- Intermediate data that has been transformed.
│   ├── processed   <- The final, canonical data sets for modeling.
│   └── raw         <- The original, immutable data dump.

├── models    <- Trained and serialized models, model predictions, or model summaries
│
├── notebooks   <- Jupyter notebooks. Naming convention is a number (for ordering),
|                 the creator's initials, and a short `-` delimited description, e.g.
|                 `1.0-jqp-initial-data-exploration`.
│
├── references    <- Data dictionaries, manuals, and all other explanatory materials.
│
├── reports         <- Generated analysis as HTML, PDF, LaTeX, etc.
│   └── figures     <- Generated graphics and figures to be used in reporting
│

├── requirements.txt    <- The requirements file for reproducing  the analysis environment.
│
├── src                <- Source code for use in this project.
│   ├── data           <- Scripts to download or generate data
│   │   └── make_dataset.py
│   │
│   ├── features       <- Scripts to turn raw data into features for modeling
│   │   └── build_features.py
│   │
│   ├── models         <- Scripts to train models and then use trained models to make
│   │   │                 predictions
│   │   ├── predict_model.py
│   │   └── train_model.py
│   │
│   └── visualization  <- Scripts to create exploratory and results oriented visualizations
│       └── visualize.py

Data is Immutable

Don’t ever edit your raw data, especially not manually, and especially not in Excel. Don’t overwrite your raw data. Don’t save multiple versions of the raw data. Treat the data (and its format) as immutable.

The code you write should move the raw data through a pipeline to your final analysis. You shouldn’t have to run all of the steps every time you want to make a new figure (see Analysis is a DAG), but anyone should be able to reproduce the final products with only the code in src and the data in data/raw.

Also, if data is immutable, it doesn’t need source control in the same way that code does. Therefore, by default, the data folder is included in the .gitignore file

Analysis is a DAG

Often in an analysis you have long-running steps that preprocess data or train models. If these steps have been run already (and you have stored the output somewhere like the data/interim directory), you don’t want to wait to rerun them every time. We prefer make for managing steps that depend on each other, especially the long-running ones.

• This will be the point of emphasis in using targets, bringing Make-like functionality to R.

Functional Programming

It’s hard to describe exactly what a functional style is, but generally I think it means decomposing a big problem into smaller pieces, then solving each piece with a function or combination of functions. When using a functional style, you strive to decompose components of the problem into isolated functions that operate independently. Each function taken by itself is simple and straightforward to understand; complexity is handled by composing functions in various ways.

• CookieCutter Data Science doesn’t go into much detail on what your src code should look like, but I have found it naturally suits a functional programming style.

• Rather than writing scripts that execute tasks, it’s generally better to write a series of functions that are then called and used in a pipeline.

A Repo Should Be One Project

Another thing that Cookie Cutter Data Science helps address: what should even be a repo? When we’re working on a project, how do we define and organize our code?

Do we create one repository for all of our data science projects? Do we create one repository per project?

Monorepo vs Multi-repo

A monorepo is one repository that contains code for a lot of different projects and tasks.

Imagine you have one big project you’re working on, containing a lot of separate pieces and code. The monorepo approach says, throw it all in into the same repo.

Monorepo vs Multi-repo

As opposed to a multi-repo, where aspects of a larger project are isolated and separated into individual repositories.

##

A Repo Should Be One Project

The Cookie Cutter Data Science approach is much more conducive towards the multi-repo approach:

• A repository exists for a specific task.
• The code in the repository executes that task.
• The requirements for running that code are defined in the repository.

But what if we want to re-use code across multiple repositories!

CookieCutter Data Science (for R)

Given these principles, most of my repos end up being organized in the following way:

├── _targets    <- stores the metadata and objects of your pipeline
├── renv        <- information relating to your R packages and dependencies
├── data        <- data sources used as an input into the pipeline
├── src         <- functions used in project/targets pipeline
|   ├── data    <- functions relating to loading and cleaning data
|   ├── models    <- functions involved with training models
|   ├── reports   <- functions used in generating tables and visualizations for reports
├── _targets.R    <- script that runs the targets pipeline
├── renv.lock     <- lockfile detailing project requirements and dependencies

renv

Let’s go back to the issues we had in running certain files in the starwars or board_games repo.

The renv package aims to solve most of these problems by helping you to create reproducible environments for R projects.

renv allows you to scan and find packages used in your project. This produces a list of packages with their current versions and dependencies. Using renv with a project adds three pieces to your repo:

• renv/library: a library that contains all packages currently used by your project.

We then add (pieces) of renv/library, renv.lock, and .Rprofile to our repository and commit them.

So, how do we do this?

We will need to get to know a few functions from renv.

renv key functions

• renv::init() initializes renv in a project. This will create a scan for dependencies, install them in a project library, and create a lockfile describing the current state of the project.

• renv::restore() restores a project’s dependencies from a lockfile. This is typically the first command when working with a repo that has an existing lockfile.